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Purpose and Objecti ve 

The Department is ultimately responsible for the care, safety and protection of childre 
pursuant to Chapter 39, Florida Statutes. 

This is the second in a series of audit and consulting reports which takes a high-level look at t 
model used for outsourcing child welfare services, and will focus on whether effective oversig 
has been established. 

The objective of this audit is to determine whether the current structure of the community- 
based care model ensures adequate oversight in the provision of foster care and rela 
services. 



June 25, 2008 


Scope and Methodolo gy 

The scope of this audit addresses the following oversight functions: 

• Contract Management 

• Contract Monitoring 

• Fiscal Monitoring 

• Quality Assurance 

• Performance Measures 

To meet our audit objective, we performed the following: 

• Reviewed governing laws; 

• Reviewed applicable audits and evaluations conducted by the Office of Program Policy 
Analysis and Government Accountability, the Auditor General, and other sources; 

• Interviewed appropriate Department Management and staff; 

• Interviewed community-based care lead agency management; 

• Obtained an understanding of the Department's oversight functions; and, 

• Performed additional testwork, as deemed necessary. 
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Background 

As pointed out in Office of Inspector General Audit Report #A-05-0708-070, under the 
Agency Model of community-based care, the Department generally does not have a d 
relationship with (i.e., does not contract with or directly oversee) most of the entities wl 
actually provide services to the children. Lead agencies are allowed, with some exceptions, t 
enter into subcontracts for child welfare services without obtaining prior Department v 
approval, and lead agencies are given the responsibility for monitoring and quality assurance 
these subcontracts. 

This poses an inherent risk to the Department, because it must rely on the lead age 
provide assurancesthat the Departmentis in compliancewith its legal and fiduciary 
responsibilities for the care, safety and protection of children. In this report, the Offf 
Inspector General will address the effectiveness of the Department's oversight process 
assuring that the children are being well-served under outsourcing. 

A future Office of Inspector General consulting report will be examining the model used for ch 
welfare service delivery by both the Department and providers. 


Contract Manageme nt 

Under s. 287.057(15), Florida Statutes (F.S.), the contract manager is responsible for enforcin 
performance of the contract terms and conditions and serve as a liaison with the contractor. 
In 2005, the Legislatureenacted Chapter 2005-222,Laws of Florida (LoF), to remove 
uncertainty as to the appropriate role and authority of the Department in providing oversight 
outsourced services. The law created s. 402.7305, F.S., which requires the Departmei 
establish a contract management process and requires the Department's contract man 
perform specific tasks. 

As stated above, the contract manager is responsible for enforcing contract compliance. The 
contract manager is also the primary point of contact between the Department and il 
contractors, and is responsible for the day-to-day contract oversight. When appropriate, the 
contract manager is required to request corrective actions, and to review, approve and enforc 
corrective action plans of providers. The contract manager is also required to follow-up with 
the provider to ensure implementation of corrective actions. Documentation of approv 
corrective action plans, status reports (as required) and completion of the corrective action pi 
is required to be maintained in the contract file. 

The Department publishes CF Operating Procedure No. 75-2, "Contract Management System 
Contractual Services," which includes a chapter providing procedures and guidance regarding 
the responsibilities of the contract manager. However, some Department administrate 
expressed concerned that administrative cutbacks were preventing community-based Cc 
contract managers from effectively carrying out their responsibilities. 
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As one administrator said: 

"The Department has abridged its responsibilities by not having quality contract 
management and supervision. We need contract managers who understand both 
program and contract management." 

Section 402.7305, F.S., requires the contract manager to periodically visit the physical locatio 
where the services are delivered and speak directly to clients receiving the services c 
staff responsible for delivering the services. Under the Lead Agency Model of community-bas 
care, compliancewith the statute generally necessitatesthe contract managervisiting 
subcontractor sites where much of the actual service delivery is occurring. However, based c 
interviews with Department administrators and staff, this may not be taking place: 

• "You do not get direct input as to the quality of the subcontractors' work. We 
have to have some direct way of knowing what they are doing. The laye 
prevent you from reaching out directly to the subcontractors and getting action." 

• "It's difficult to pierce the subcontractor veil." 

• "There are no surprise visits, nor is it routine to visit the subcontractors." 

• "The lead agency is responsible for overseeing their subcontractors.... Even if the 
lead agency agreed to more visits by us, what authority would we have when we 
are out there?" 

Furthermore, under s. 402.7305, F.S., it is the contract manager's responsibility to periodically 
documentany differencesbetweenthe required performancemeasuresand the actual 
performance measures. If a provider fails to meet and comply with established performance 
measures, the Department must terminate the cohtrwelyer, the Department may consider 
extenuatingcircumstancesand allow a reasonable period for the provider to correct 
deficiencies. Performance measures will be discussed as a separate topic in this repc 
However, some Department administrators and staff have expressed concerned over the tim< 
availability and reliability of performance data from the Department's automated child welfar 
information system - the Florida Safe Families Network (FSFN): 

• "There have been a lot of issues with the quality, accuracy and frequency 
information in FSFN. I would say that we are reviewing the data due to possible 
errors in the system." 

• "... We can't get the necessary data out of FSFN." 

While the Department has established a contract management process, the Lead Agency Moc 
presents unique challenges in the oversight of outsourced services. As a result, community- 
based care contract managers may need to take a more active oversight role which includes 
periodic visits to subcontractors, talking with staff and children, and observing the delivery of 
services. 
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Contract and Fiscal Monitor ing 


Contract Monitorin g 

Section 402.7305, F.S., also requires the Department establish contract monitoring units and 
contract monitoring process which includes performing onsite monitoring at external location: 
a service provider. The law states, in pafTThe department may monitor any contract at am 
time...."Contract monitoring is conducted primarily by the Department's Contract Ove 
Units. The statute describes contract monitors as being responsible for observing, recording 
and reporting the information necessary to assist contract managers and program manager™ 
in determining whether providers are in compliance with the terms and conditions of 
contract. Contract monitoring is not intended; however, to assess quality of service delivery. 

Since 2005, responsibility for monitoring contracts has gone through several organizati 
changes, having been assigned to the former 13 districts and one region, the Office of Inspec 
General, the Office of Quality Management and, in fiscal year (FY) 07/08, the Office c 
Assistant Secretary for Programs. While under Quality Management, an integrated rev 
process was used in which Contract Oversight, Quality Assurance and Licensure staff worked 
together to conduct a single review of community-based care lead agencies. Flowever, under 
the Assistant Secretary for Programs, Contract Oversight has conducted separate monitoring 
lead agencies, and the Chief of Contract Oversight does not expect the integrated re 1 
process to continue. 

Contract Oversight Units are responsible for monitoring the contracts with each of the 
community-based care lead agencies, except for ChildNet and Our Kids. Notwithstand 
s. 402.7305, F.S., in 2006, the Legislature passed Chapter 2006-30, LoF, which, under a three 
year pilot program, transferred fiscal, administrative and programmatic monitoring of Chi IdNe 
and Our Kids tdindependent, nongovernmental third party ©afetebfesoh discussions 
with the General Counsel'soffice, Chapter 2006-30, LoF, is specific to monitoringthe 
community-based care contracts with ChildNet and Our Kids, and takes precedent ove 
Department's general authority under s. 402.7305, F.S. to monitor any contract. The reason 1 
the transfer of oversight responsibility and its impact will be discussed further throughout this 
report. 

Under the Lead Agency Model, lead agencies are responsible for monitoring subcontracts whi 
they enter into with entities that provide direct services to the children. Lead agencies are al 
responsible for developing written procedures for monitoring of subcontracts. Accordin 
Contract Oversight staff, contract monitoring of lead agencies includes reviewing the I 
agency's monitoring schedule, monitoring procedures, reports issued and timelinessof 
corrective action to report findings. However, it does not include visits to subcontrac 
unless, in their determination, the lead agency fails to monitor its subcontracts sufficiently. 
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In accordance with s. 402.7305, F.S., the Department has established contract monitoring uni 
and a contract monitoring process. Furthermore, in a recent progl'efetser£)tf6de of 
Program Policy Analysis and Government Accountability (OPPAGA) found that the Department 
has taken steps, consistent with OPPAGA's recommendations, to strengthen its contrac 
oversight system, including the contract monitoring process. 

Fiscal Monitorin g 

The Department's FY 07/08 standard lead agency contract Attachment I describes fisc 
monitoring as providing financial oversight and ensuring the integrity of the provider's 
operations, including adherence to generally accepted accounting principles and the appropri 
use of various funding streams. 

As previously mentioned, Chapter 2006-30, LoF, established a three-year pilot program which 
transferred the Department's fiscal, administrative and programmatic oversight of the 
agencies in Broward, Miami-Dade and Monroe Counties to independent third party entities. T 
was at the request of Our Kids management which, in their words, sought independence from 
the Department and questioned the competence of the Department's monitors. 

In response to Chapter 2006-30, LoF, the Department contracted with Abel & Associates, PA, 
CPAs in August 2006, to provide fiscal and administrative monitoring (including tasks previou: 
performed by Contract Oversight Units) of ChildNet and Our Kids. The law requires that the 
pilot program not be implemented until the parties agree to the selection and manner in whic 
the monitor would carry out its responsibilities. According to the Auditor General, Ab 
Associates was awarded the contract because they were already providing satisfactory financ 
monitoring of ChildNet for the Department since October 2003, and Our Kids also agreed with 
the selection. 

Although not required by Chapter 2006-30, LoF, former Secretary Lucy Hadi directed that fisc 
monitoring of the other lead agencies also be outsourced, and the Department contracted wil 
Public Consulting Group (PCG) to be fiscal monitor. However, by the end of December 2007, 
PCG's contracts were terminated and a decision made to bring in-house fiscal monitor 
(excluding ChildNet and Our Kids). According to the Assistant Secretary for Administn 
moving fiscal monitoring in-house will enable the Department to more quickly identify 
address fiscal issues as they arise. She indicated a desire to have in-house fiscal monitoring 
begin July 1, 2008; however, budgetary restrictions may delay implementation. 

With regard to the impact of Chapter 2006-30, LoF, on fiscal monitoring, Office of Inspector 
General consulting report #C-05-07-08-026 states,Mowing contractors to be part of th 
selection process and establishing the manner in which they are monitored does not pr< 
independence and accountability." 


1 Source: OPPAGA Report No. 08-32, "Department of Children and Families Has Strengthened Its Contract Oversight System," published Ma> 
2008. 

2 

Auditor General Report No. 2008-072, "Department of Children and Family Services - Community-Based Care Pilot Program - Fiscal and 
Administrative Monitoring - Operational Audit,” published January 2008. 
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Recommendatio n 

In light of recent improvements in contract oversight, and to enhance accountability, 
recommend that at the conclusion of the three-year pilot program, the Department propose 
that it retain responsibility for fiscal and administrative monitoring of all lead agencies. 

Management Respon se 

"This recommendation will be taken into account when the Department makes its proposal to 
the Legislature. Because the pilot is not yet completed, it would be premature to make a fina 
decision at this time." 


Quality Assuranc e 

Title 45, Chapter XIII, Part 1355.34(c)(3), Code of Federal Regulations, requires states 
operate an identifiable quality assurance system. The quality assurance system is one of the 
seven systemic factors assessed through the Federal Child and Family Services Review (CFSR 
process. This review will be discussed further in the next section, Performance Measures. 

Quality assurance revievVassess the quality of services provided to children and far 
determine whether case work services are comprehensive and services received 
clients'needs,and evaluatewhethercase workers'decisionswere in the clients'best 
interest'^. 

Although the Department has had some form of quality assurance relative to child protective 
services since 19^9;Jiis inspector general report focuses on quality assurance for community- 
based care. In 2001, OPPAGA reported that the Department's system for monitoring provider 
did not allow for routine assessment of the quality of the provider's service pr6vSraxre 
then, development and implementation by the Department of a viable quality assurance syst 
for community-based care has gone through several revisions: 

• OPPAGA reported that the Department piloted an integrated monitoring review ol 
Sun Coast Region in September 2002, as well as a System of Care review. However, 
they added,"... the program's integrated monitoring approach has not stream 
current duplicative and overlapping monitoring processes. 

• The 2003 Legislature appropriated 88 new FTEs to the Department to expand its quality 
assurance function. These positions were outposted to the [former] district offic 


3 Source: OPPAGA Report No. 08-09, "Pilot to Outsource CBC Program Oversight Encountered Setbacks; Effectiveness Unknown," published 
February 2008. 

4 Source: "Florida Statewide Assessment," prepared in conjunction with Florida’s first CFSR, August 2001. 

5 Source: OPPAGA Report No. 01-14, "Justification Review - Child Protection Program,” published March 2001. 

6 Source: OPPAGA Report No. 03-09, "Progress Report: Child Protection Program Makes Needed Changes But Lacks Data for Evaluating Resu 
of Initiatives," published January 2003. 
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OPPAGA reported that the Department planned to implement a new three-tiered quality 
assurance/quality improvement system in early 7 * 9 10 2004. 

• The Department implemented the Child Welfare Integrated Quality Assurance (CWIQA) 
review process, and over 2004 and 2005, CWIQA reviews occurred in every leac 
agency? The statewidetransition to community-baseotare was completed in 

April 2005. 

• In July 2005, the Office of Quality Management was created, and charged with 
establishing a Department-widequality assurance process. The Department 
implemented a three-tiered quality assurance system beginning in April 2006. Under tl 
system, each lead agency was responsible for developing and implementing a Q 
Management Plan for reviewing in-house and subcontracted services (Tier 1), wh 
Department staff were responsible for approving lead agency plans and validating lead 
agency quality assurance reviews (Tilr 2). 

To enable lead agencies to carry out Tier 1 quality assurance responsibilities, in 
July 2006, the Department eliminated roughly half (43 FTEs) of its quality assurance sta 
statewide and transferred these funds to lead ag^cies. 

• In late January 2007, Secretary Butterworth created an Organizational Review Wc 
Group, which recommended the Department go to a regional structure. As a re 
Department quality assurance staff were moved out of the Office of Quality Managemei 
and placed under the direction of regional management. In addition, a Child Welfare 
Quality AssuranceRegional Model was developed,approved by the Secretary in 
March 2008, and is expected to be implemented on July 1, 2008. 

The Regional Model incorporates a methodology used by the Chapin Hall Center 
Children (Chapin Hall). It eliminates the three-tiered model, replacing it with base, side 
by-side, and in-depth quality reviews, in addition to 100% case management supervisoi 
reviews. An Implementation and Oversight Team, comprised of Department and lead 
agency management, has been chosen to make decisions as to the specifics of the acti 
implementation, including the statewide minimum criteria for the community-based car 
lead agencies' quality assurance plans for FY 08/09. 

Pursuant to Chapter 2006-30, LoF, programmatic monitoring (including quality assuranc 
ChildNet and Our Kids is to be conducted by an independent third party entity. In response tc 
the law, the Department contracted with Chapin Hall (through the University of Chicat 
November 2006 to conduct program monitoring of ChildNet and Our Kids through qua 


7 Source: OPPAGA Report No. 04-03, "Special Report: DCF Needs to Improve Child Protection Staff Training and Clarify DCF and Lead Agency 
Roles," published January 2004. 

o 

Source: "Statewide Assessment Instrument," prepared in conjunction with Florida’s second CFSR, January 2008. 

9 Source: OPPAGA Report No. 07-03, "CBC Pilot Project Implementation Delayed But Proceeding: Other Initiatives Implemented," published 
January 2007. 

10 Source: OPPAGA Report No. 07-03, "CBC Pilot Project Implementation Delayed But Proceeding; Other Initiatives Implemented," published 
January 2007. 
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assurance reviews and to develop new performance outcome measures. According to OPPAG 
Chapin Hall was selected at the recommendation of ChildNet and Our Kids mafegement. 

OPPAGA's interim evaluation ofthe pilot program found that implementatiorlnas been 
substantially delayed, in part, because Chapin Hall used the Department's quality asst 
system. As described in OPPAGA's evaluation, problems with the quality assurance s^ 
hindered Chapin Hall's ability to successfully implement program oversight, resulting in a lap? 
of almost two years in state quality assurance reviews for the pilot lead agencies. O 
further stated that Chapin Hall proposed changes to the quality assurance methodology, whic 
the Department accepted and incorporated into the Regional Model (referred \b above). 

The delay was also caused by ChildNet and Our Kids disagreement with the accuracy of Chap 
Hall's draft July 2007 report. The lead agencies and the Department questioned the approacf 
used by Chapin Hall's subcontractors. Chapin Hall agreed not to finalize the July 2007 review 
and instead submitted a revised monitoring report in September 2007, which was accepted b 
the Department and lead agerl^iein April 2008, Chapin Hall submitted a draft of their 
second quarter monitoring report for ChildNet and Our Kids, and, with regard to Our 
observed, "The findings from this case record review support the findings reported as pai 
the prior review." 

Although the statewide transition to community-based care was completed in April 2005, the 
Department's quality assurance system continues to go through revision. Without a v 
quality assurance system in place for community-based care, the Department is unabl 
effectively assess the quality of outsourced child welfare services. 


Performance Measur es 

Another factor in accountability is the set of performance measures used to set standards, fo< 
improvement efforts, and evaluate success. A measure may be defined as a quantitative or 
qualitative indicator of the extent to which objectives are being achieved. Based on interviev 
with staff of the Office of the Assistant Secretary for Programs, the Department has various s< 
of measures for outsourced child welfare services; however, they are not fully aligned with ea 
other: 

• The Department'^ 07/08 standard, and system of care lead agency contract 

Attachment I's include eight performance measures, four of which are consistent 
Federal measures. The other four were internally created, based either on priorities of 
previous Department heads, or measures which lead agencies can control. These eight 


11 Source: OPPAGA Report No. 07-03, "CBC Pilot Project Implementation Delayed But Proceeding; Other Initiatives Implemented," published 
January 2007. 

12 Source: OPPAGA Report No. 08-09, "Pilot to Outsource CBC Program Oversight Encountered Setbacks; Effectiveness Unknown," published 
February 2008. 

13 

Source: OPPAGA Report No. 08-09, "Pilot to Outsource CBC Program Oversight Encountered Setbacks; Effectiveness Unknown," published 
February 2008. 
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performance measures are also used in the pilot program contracts with ChildNet and 
Our Kids. 

• The Department's Long-Range Program Plan (LRPP) specifies legislatively approvei 
performance measures and standards for each program area. However, only five of the 
contract performance measures (including the four Federal measures) are included in tl 
LRPP for the Child Protection and Permanency budget entity. 

• The February 26, 2008 edition of the DCF Quick Facts identifies "Critical Few 
Performance Measures." Five of the six critical few measures pertaining to outsourced 
child welfare services are included in the lead agency contracts. However, thib 4 sixth, 
which is based on a Federal measure, is neither in the contract or the LRPP. 

Furthermore, none of these sets of measures may be entirely meaningful, or consistent with 
recent changes in Federal measures: 

• Chapter 2006-30, LoF, reguires the contracted third party monitor (Chapin Hall) develop 
other performance outcome measures to best determine the guality of performance of 
the lead agencies. Chapin Hall is contractuallyreguired to report on their 
recommendations for other performance measures by August 1^, 2008. 

• The Federal Child and Family Services Review (CFSR) monitors the state's confor 
with the reguirements of Title IV-B of the Social Security ^cf.lorida's second CFSR 
occurred in January 2008. Areas not in substantial conformity will reguire a Proc 
Improvement Plan, and, if necessary, significant financial penalties will be assessed for 
those areas not achieving the necessary improvements. 

In addition to seven systemic factors, the CFSR measures seven outcomes addressing 
safety, permanency, and child and family well-being. In 2007, the Department adaptec 
a set of 21 lead agency contract measures based on specific measures used fo 
CFSR. Although the Assistant Secretary for Programs considers the measures "solid", h 
stated that existing lead agency contracts will not be amended, at this time, to reflect 
these new measures. Instead, the Implementation and Oversight Team is to come up 
with recommendations on how to track and monitor compliance with the new Federal 
measures. The Assistant Secretary for Programs tentatively expects the Team to have 
these recommendations by October 1, 2008. 

The Department has various sets of measures for outsourced child welfare services; however 
they are not fully aligned with each other, and none of the sets may be entirely meaningful oi 
consistent with recent changes in Federal measures. As a result, the Department may lack th 
relevant information needed to effectivelymeasure safety, permanencyand well-being 
outcomes for children and their families receiving child welfare services. 


14 Percentage of children with no more than two placements within 12 months of removal. 

15 Contract JJ210, as amended in November 2007. 

16 Source: The Federal Administration for Children & Families website. 
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Recommendatio n 

We recommend the Department take prompt action by adopting a uniform set of performan 
measures for outsourced child welfare services which meet Federal requirements, and holdinc 
lead agencies contractually accountable for these measures. 

Management Respon se 

"Upon approval of the Child Welfare Quality Assurance Model on March 11, 2008, the Family 
Safety Program Office convened a Performance Measure workgroup to review the existing set 
state, federal, contract and best practice measures and methodologies. One of the overarchi 
objectives of the workgroup is to develop a set of measures that drives practice consistent wi 
Florida's goals and can be used for CBC contracts and Quality Assurance/Quality Improvemen 
The workgroup is comprised of representatives from the Department (Office of the Secretary, 
Central Office and Region Family Safety staff), CBC's (Executive Directors and QA staff) and tf 
University of South Florida, Florida Mental HeaIth Institute. The workgroup plans to have this 
set of measures developed and ready for inclusion in CBC contracts beginning July 1, 2009." 
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This report is available on our website: http://www.dcf.state.fl.us/admin/ig/pubs_ia.shtml 


10 






